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ADDENDUM 


TO  REPORT  ON  THE 
FINAL  ACCEPTANCE  TEST  FOR  CTS 

During  the  period  from  2  August  1976  through  24  September  1976,  action 
was  taken  by  the  contractor,  GTE-Sylvanla,  Inc.,  Needham  Heights,  MA, 
and  US  Army  personnel,  CTS  Project,  Fort  Gordon,  GA,  to  correct  the 
deficiencies  and  problems  found  during  the  Phase  III  Acceptance  Test 
described  in  this  report. 

Verification  tests  were  performed  over  the  period  from  27  September  1976 
through  8  October  1976.  All  corrections  to  the  problems  and  deficien¬ 
cies  cited  were  completed  to  the  satisfaction  of  the  Army  test  personnel 

The  delivery  of  the  final  report  completed  the  contract  requirements  and 
the  system  was  accepted  for  the  government  by  the  Contracting  Officer's 
Technical  Representative  on  22  October  1976. 
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1.0  Purpose 

This  report  describes  the  conduct  and  results  of  the  Phase  III  Acceptance 
Test  for  CTS.  This  test  was  conducted  on  February  23-27,  March  22-23,  and 
July  19-30,  1976  at  Fort  Gordon.  The  purpose  of  this  report  is  to  provide 
an  objective  reference  document  for  the  use  of  the  U.S.  Army  in  making  de¬ 
cisions  on  the  acceptance  of  the  CTS  system  and  related  contractual  matters. 

^2.0  Test  Objectives 

The  primary  objectives  of  the  test  were  to  determine  whether  the  full 

128-terminal  system  will  operate  under  operational  live  user  conditions  and 

meet  contract  requirements  for  reliability  and  response  time.  -.The  ability 

— ■ ?  to  m 7% 

of  the  system  hardware  and  software  to  support  individual  user  functions  had 
been  tested  in  Phases  I  and  II. 

3.0  Test  Plan 

Detailed  procedures  for  the  test  are  described  in  Appendix  A:  Phase  III 

CTS  Final  Acceptance  Test  Plan.  The  test  was  conducted  in  five  levels,  corres 

ponding  to  five  test  objectives,  as  follows: 

Level  I:  Verification  of  Phase  II  Software  Tests 
Level  II:  Reliability 

Level  III;  Verification  of  Phase  II  Class  I  Language  Tests 

Level  IV:  Response  Time 

Level  V:  Full  Operational  Load 

t 

4.0  Tost  Results 

The  results  of  the  tests  are  described  below  with  reference  to  the  re¬ 
quirements  for  Phase  III  tests  listed  in  modification  no.  P00006  to  Contract 


Mo.  DAHC26-74-C-0006 . 


4.1.0  Verification  of  Phase  II  Results:  "Verify  that  the  system  software 
performs  at  the  same  level  as  when  Phase  II  acceptance  tests  were  conducted, 
kt  preservative  tests  will  be  taken  from  the  Phase  II  ATS  and  run  to  obtain 
verification  " 

4.1.1  Results:  On  February  23  and  24,  1976,  test  procedures  for 
Levels  I  and  III  were  followed.  These  tests  verified  that  System  Readiness, 
system  Programmer  functions,  and  the  Class  I  Language  commands  operate  sat-- 
!.;f  '.ct-vrlly  (with  exceptions  noted  In  Section  5.1.0  of  this  report'1. 

A.I.O  Reliability :  "(4.1.2)  The  system  must  achieve  a  minimum  of  95% 

vu-tLno  of  the  use  time  run  10  days  of  tests,  using  the  running  time  meter, 
'.ip-time  refers  to  the  capability  of  the  hardware  and  software  to  perform  basic 
ask  ■  of  system  initiation,  program  compilation,  program  execution,  job  lr-mago- 
-  t,  file  management,  and  program  library  maintenance." 

4.2.1  Method  for  Testing  Reliability:  "(4.1.3)  The  Army  will 

•  rovide  the  contrator  with  a  test  course  to  be  used  in  the  acceptance  test." 
"(4.3. 3.1)  The  primary  method  of  testing  the  full  operational  performance 

>  i  the  system  shall  be  by  extensive  use  of  the  system  by  students,  instructional 
P rogr  i.'jQars,  and  administrators  using  an  Army-provided  test  course." 

It  was  not  feasible  for  the  Army  to  provide  128  live  users  for  a  ton 
day  period  to  test  reliability  of  the  system  under  conditions  of  actual  use. 
heroiore  a  simulator  program  was  run  during  65  hours  of  the  test. 

The  simulator  program  consisted  in  a  special  Class  I  lesson  that 
•••luomati  tally  a. id  randomly  presents  a  variety  of  displays  over  a  range  of 
arn  h'  to  40  seconds  between  displays.  This  lesson  can  execute  unattended 
-V'-e  it  la  initiated  at  a  terminal.  The  set  of  random  time  intervals  is  gen- 

•  c’d  .it  the  start  of  the  lesson  and  is  unique  each  time  the  lesson  is 


executed.  The  lesson  will  cycle  indefinitely  and  will  only  stop  at  lob  off. 
The  simulated  load  condition  is  achieved  by  run/ing  this  automatic  display 
program  asychronously  at  all  terminals. 

When  the  simulator  program  is  operating  with  all  128  terminals,  all 
hardware  components  of  the  system  are  in  operation.  Thus,  the  simulator  pro¬ 
vides  a  cinvenient  and  adequate  method  of  testing  hardware  reliability. 

This  particular  simulator  program  is  not  adequate  for  testing  soft¬ 
ware  reliability,  however.  The  program  does  not  generate  all  the  software 
conditions  that  are  encountered  in  full  operational  use  with  live  users. 

4.2.2  Results  of  Reliability  Teats:  The  system  achieved  over  95% 
up-time  during  the  65  hours  in  which  the  system  was  exercised  by  the  simula¬ 
tor  program.  When  full  operational  performance  of  the  system  was  tested  with 
students,  instructional  programmers  and  instructors,  the  reliability  was  less 
than  50%.  j 

Thus,  the  simulator  program  was  demonstrated  to  be  an  inadequate 
test  of  system  software  reliability.  The  system  was  found  not  to  meet  relia¬ 
bility  requirements  under  the  conditions  of  the  contract  as  6tated  in  paragraph 
4.2.1  above. 

4.3.0  Response  Time:  Performance  specifications  for  the  128  system  state 
that  there  shall  be  a  90%  probability  that  student  response  time  shall  be  2 
seconds  or  less.  No  requirement  is  stated  for  Instructional  Programmer 
response  time. 

4.3.1  Method  for  Testing  Response  Time:  According  to  modifier*-4^ 
n^.  P00006  to  Contract  No.  DAHC26-74-C-0006,  paragraph  4.1.4,  "The  pMlft” 
of  the  system  to  meet  response  time  requirements  will  be  tested  through  t!.-' 
simulator  program  that  will  measure  performance  under  varying  input  loadc. 
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"he  response  times  will  also  be  tested  through  use  of  the  system  with  a 
V  :  complement  of  users  on  the  128  terminal  system." 

T'he  simulator  program  described  in  paragraph  A. 2.1  was  used  in  ccr  - 
Sa^i'.hm  with  several  increments  of  live  users  to  test  response  tires  fV  -a 
i.t  v  L  'i  of  the  Acceptance  Test  Plan,  impend ix  A) .  The  procedure  is  to 
-iru'lo  onto  the  simulator  program.  Then  a  epe.ci.ficd  number  v  ~  the 
,  i:  ..is  logged  off  the  simulator  and  legged  on  by  Jive  students,  in  - 

■:i  :  instructional  programmers.  The  live  users  thus  "compete"  rv..i 
■  :  users  for  response  time.  The  live  user  response  times  wore  wb* 

-■  ■  :  .■  -  !  ?..! mud  as  they  interact  with  the  system.  Response  tire  is  defined 
■  m->  elapsed  between  pressing  the  "NEXT"  key  and  a  change  on  the 


r  i.uvvJ  V,  a  full  complement  of  128  live  users  was  planned  to  re- 
■  times  under  non-simulator  conditions.  It  was  not  possible  to 
•  • , r  ■  ri  .e  times  under  this  condition  because  of  system  and  Display 

•  •  i  ■  i-  > 

!i  i.  he  simulator  program  does  not  adequately  simulate  opern- 
.  i,  i:ul  because  response  recordings  could  not  bo  taken  under 
r  >im1  conditions,  the  response  tire  results  should  no*-  be 
•.»:  .  est.  of  response  time. 

• *1<  cults  of  Response  Time  Tests:  Response  time  recordings 
,  f  hr. 'e-hour  period  on  July  27.  under  conditions  described  in 
•he  •u.-cepCHitce  Test  Plan.  Between  A  and  6  live  students  and  one 

•  •  j  <  (1 A-20  overall)  were  operating,  with  the  remaining  termin'.  1  s 

•  :  r  -,fr  1 1  at  ion  trK.de.  Under  these  conditions  a  total  of  191  rc~ 

r  ■  nv:.i  ’  <•;  student  response  time.  The.  results  were  as  follows: 


—A  — 


-  154,  or  80%,  of  the  student  response  times  recorded  were 

1  second  or  los3. 

-  22,  or  12%,  of  the  recordings  were  between  1  and  2  seconds. 

-  15,  or  8%,  of  the  recordings  were  greater  than  2  seconds. 

During  portions  of  the  test,  from  1  to  2  Instructional  Programmers  operated 
per  Display  Controller  (4-8  Instructional  Programmers  total  per  system) .  With 
one  Instructional  Programmer  per  DC,  the  average  response  time  recorded  was 
11  seconds,  with  a  range  of  from  1  to  24  seconds.  With  2  Instructional  Pro¬ 
grammers  per  DC,  the  average  was  43  seconds  response  time  with  a  range  of  1 
second  to  2  minutes. 

The  test  plan  calls  for  response  time  recordings  under  two  additional 
conditions:  with  a  background  batch  job  and  with  a  full  load  of  128  live  users. 
It  was  not  possible  to  obtain  response  time  recordings  under  these  conditions 
because  the  system  would  not  operate  reliably  enough  under  these  conditions  to 
perform  the  tests. 

4.3.3  Discussion  of  Response  Time  Results:  Due  to  unreliability  of 
the  system  under  conditions  of  operational  use,  it  was  not  possible  to  obtain 
a  sufficient  number  of  response  time  recordings  to  ensure  that  a  representative 
sample  had  been  made.  Therefore  we  are  unable  to  state  with  confidence  whether 
or  not  the  system  meets  response  time  requirements.  Army  instructional  pro¬ 
grammer  personnel  who  have  been  using  the  system  to  develop  and  debug  instruc¬ 
tional  programs  have  reported  unacceptable  delays  in  response  time  in  instruc¬ 
tional  programmer  mode.  No  complaints  have  been  voiced  about  response  time  in 
student  or  instructor  modes. 
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5.1.0  The  Army  CTO  Field  Office  and  the  Signal  School  at  Fort  Gordon 
have  been  using  the  system  Cor  Instructional  Programmer  activities  for  sevc/i 
'’onth.:i.  Daring  this  period  they  have  been  diagnosing  and  debugging  the  system, 
ub  .•  assistance  from  the  contractor.  Lt.  David  Wicker t,  the  CIS  Field  Off; 

■  -I-:  v*  pro,-;  rammer,  has  listed  specific  problems  that  have  been  recently  di  •  ;r 

.  "■.••t  corrected.  These  are  as  follows: 

5.1.1  DBG  Addition  Algorithm  of  Lesson  Units:  The  DBC  assigns  too 

:■  a  •  ira  in  its  lesson  unit  disk  usage  (LUDU)  table  for  a  lesson  unit.  A  ur>  *  t 
which  was  6  blocks  took  up  12  blocks  in  the  LUDU  BITMAP.  This  results  in  beta 
1  v-asjiented  disk  and  wasteful  space  on  the  disk.  Software  Systems  Element:  TT 
7  •"  aritua,.  States:  GTE -Sylvan ia  is  investigating  the  problem. 

5.1.2  User  Mode  TIB  Execution:  The  TIB  of  the  DC  executive  now 

o --acutes  in  Kernel  mode  instead  of  user  mode.  Reference  MFR,  ATTN G-TA-T S-A , 

•  Fan  76,  sab;j:  Removal  of  a  DC  Executive  Protection  Feature  (Tncl  1).  Soft- 
. -r  Systems  Element:  DC  Executive.  Status:  Problem  has  not  been  addressed 
,-v  G !  ’ .  G  •!  vania  at  this  point. 

>.1.3  DC  Executive  Grabs  DMA  Channel  to  DBC:  Under  very  heavy  trarf 
?  Dl:<  ,  there  is  timing  problem  where  the  PC  will  grab  the  channel  to  the  PPC, 

■  ”.t  has  no  means  oi  t eleasing  it.  Software  Systems  Element:  DC  Executive. 

•  t"  Solution  is  known  by  GTE -Syl vania. 

j .  j  .4  DEC  Improper  Handling  of  its  2nd  Lessor.  Unit  Output  Buffer :  PR 
i  2-2  buffers  for  handling  lesson  unit  transfers  to  either  PC's  or  SC.  Undo 
eavv  traffic  cads  or  problem  »?3,  the  DPC  is  forced  to  use  its  second  buffer. 

.  .  itrj  oiitTiit  lou.  the  2nd  buffer  is  not  used.  When  the  DBC  tries  to 


reference  its  2nd  output  buffer,  it  has  clobbered  an  important  register  used 
to  access  the  buffer.  Thus,  the  2nd  output  buffer  is  not  cleared  and  output ed 
to  DCs.  Software  Systems  Element:  DBC  Librarian.  Status:  Solution  is  known 
by  GTE-Syl vania. 

5.1.5  CLASS  1  'LOAD'  Command:  After  ASC  11  load  into  a  multiword 
buffer,  the  CLASS  1  compiler  is  required  to  space  fill  the  remainder  of  the 
buffer.  The  size  of  the  buffer  is  being  divided  by  2  before  use  so  that  only 
half  of  the  buffer  is  being  space  filled.  Software  Systems  Element:  CLASS  .1 
ccnniJer.  Status:  GTS  -  Sylvania  has  rewritten  routine. 

5.1.6  CLASS  1 'TAB 1  Command:  'TAB'  routine  in  CLASS  1  compiler  checks 
for  valid  row  number  instead  of  valid  column  number.  Thus,  ariv  column  number 

>  30  returns  an  error  of  'illegal  row  number.' 

5.1.7  DBC  Dump  of  Table  'LUlxDU':  The  DBC  librarian  includes  03 
part  of  its  directory  dump  facility,  the  capability  of  dumping  all  of  the  tables 
in  use  at  the  DBC.  All  tables  are  included  except  table  'LUlxDU.'  Although  the 
SC  does  not  provide  the  facility  to  request  end  print  out  these  tables,  they 
ore  included  in  the  unit  specification  for  DBC  (reference  page  390)  and  should 
be  complete.  Software  Systems  Element:  DBC  Librarian.  Status:  Solution  is 
known  by  GTE-Syl vania . 

5.1.8  Wrong  total  time  1 r  active  user  table  at  SC:  Minus  numbers 
are  in  AUT  at  SC  after  warm  start.  Software  Systems  Element:  SC  Software. 

t 

Status:  GTE-SylvanJa  is  investigating. 

5.1.9  'BACK'  command  not  being  executed  properly:  'Branch  to  last 
restart  point',  when  running  as  regular  student  is  not  going  to  DBC  to  get 
latest  restart  record  of  student.  Instead,  it  is  going  to  the  DC  disk.  Soft¬ 
ware  Systems  Element:  DC  Executive.  Status:  Solution  is  knovrn  bv  GTE-Syl  van  I  r, , 


' . 1 . 1C  CLASS  1  'Table'  command:  the  iU'*?l»*n  plan  (reference  pap.  '1 
them  are  two  differences  between  the  ’Table'  and  ’Common’  command. 

Common ’  command  must  be  first  command  in  unit,  and,  second,  th^  ,r,'ab.L>. ’ 
n.  tarines  data  internal  to  unit  only.  Both  differences  are  wrong.  i-.c, 
ntly  .implemented,  there  is  no  difference  between  the  ’ Table'  and  'Commrn' 

:•  ; .  Software  Systems  Element:  Class  1  Compiler.  Status:  GTE-Sylvan-* 
ves floating  problem. 

j .  1. .  I,  CLASS  x  command  'VANS'  Improper  tolerance  offsets:  Th  >  i  ov.er 
r.r.'Lt'  i  im*  c  on  'VAN’S’  command  is  not  being  accepted  as  within  tolerance. 

re  Systems  Element:  CLASS  1  Compiler  or  DC  Executive.  Statu:::  <7  E~ 
rin  is  investigating  problem. 

5.1.12  Editor  lost  line  number:  While  operating  the  GTS  editor,  a 
' amber  jfiset  was  missed  and  lines  were  numbered  incorrectly  from  that 

Software  Systems  Element:  SC  Editor.  Status:  GTE-Sylvania  is  investi 

'■  problem. 

5.1.13  i'odepool  Level  Problem:  The  normal  level  of  the  SC  RSX-JiD 
ooi  i:,  too  low  under  heavy  system  load.  At  fresh  reboot  of  system,  the 

w  183  nodes.  With  light  student  load,  the  nodepool  is  170  1  15  nodes 
heavy  student  load,  the  nodepool  is  1.00  ±  20  nodes.  Under  almost  ful  ’ 
'‘dents)  and  light  TP  (3  IPs)  load,  the  nodepool  is  70  ±  20  nodes.  Under 
•  indent  Load,  the  nodepool  falls  below  50  nodes.  These  levels  for  the. 

A  r odwpool  d£  no_r_  give  an  adequate  safety  margin  under  heavy  student,  rn 
ft.  ware  Systems  Element:  SC  and  RSX-11D  Software.  Status:  C7F- 
'  '  •  :  avert  I  gating  problem. 

1.1 .13  Of  Executive  Gets  Improper  SC  message:  Under  heavy  trail  V, 

■  a  ut  ;ve  aj-pears  to  get  an  Improper  £0  message  which  results  in  i  he  nq 
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PHASE  JIT  CIS  FINAL  ACCEPTANCE  IF ST  PLAN 
TEST  MONITOR  PROCEDURES  AND  LOG 

Level  III:  CLASS  I  Commands  4  hours  Moran  Hall,  Greeley  Hall,  Brant  Hall 
OBJECTIVES  OF  TEST 

1.  Ascertain  that  all  CLASS  I  commands  can  be  compiled  and  executed. 

2.  Ascertain  that  the  CLASS  T  compiler  will  recognize  and  flag  errors  during 
compilation. 


TEST  PROCEDURES 

.1  .  Three  versions  of  the  test  program  will  be  in  the  system.  One  will  be.  at 
the  Data  Base  Controller  (DBC)  with  "students"  registered;  another  will 
he  fully  debugged  source  code;  and  the  third  will  bo  source  code  with 
selected  errors  inserted. 

2.  Eight  "students"  and  four  "instructors"  divided  between  the.  four  DCs  will 
sign  on  and  execute  the  program  as  directed. 

3.  An  IP  will  sign  on  and  run  a  compile  of  the  debugged  program. 

4.  An  TP  will  sign  on  and  run  a  compile  of  the  "bugged"  program. 

r>.  Procedures  2,  3  and  4  will  be  executed  simultaneously. 

MONITOR  PROCEDURES 

1.  Terminal  monitors  for  the  four  DCs  will  observe  the  execution  of  the  test 
program  by  students  and  instructors.  As  each  CLASS  I  command  is  tested,  tb 
monitor  will  check  off  that  command  on  the  Level  TIT  log  sheet  (attached). 

2.  The  system  monitor  will  observe  the.  compilation  of  debugged  and  bugged 
urogram.  Copies  cf  the  compilation  listing  will  bo  retained  and  rnrd y.tod . 
Monitor  will  check  off  on  the  log  sheet  all  commands  and  conditions  that 
were  compiled. 
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PHASE  III  CTS  FINAL  ACCEPTANCE  TEST  PLAN 
TEST  MONITOR  PROCEDURES  AND  LOG 

Level  II:  Reliability  80  hours  Moran  Hall,  Greeley  Hall,  Brant  Hall 
OBJECTIVES  OF  TEST 

1.  Ascertain  the  system  will  meet  the  reliability  requirements  of  ?5%  up  tin; 
for  ten  8-hour  days  of  tests  using  the  running  time  meter. 

2.  Ascertain  that,  on  the  average,  all  terminals  will  operate  76  of  the  total 
80  hours . 

TEST  PROCEUDRES 

1.  The  reliability  test  will  commence  with  Level  1  and  run  concurrently 
with  test  levels  3,  4  and  5. 

2.  The  load  simulation  program  will  be  loaded  and  nil  terminals  except  those 
required  for  Levels  3,  4  and  5  vtill  be  activated  under  the  simulator. 

3.  Other  levels  of  test  will  be  conducted. 

4.  System  and  terminals  will  be  operated  for'  a  total  of  80  hours. 

MONITOR  PROCEDURES 

1 .  By stem  monitor  will  maintain  a  system  log  (example  attached!  shoving  date 
and  time  Level  II  began,  any  downtime  on  any  component,  reason  for  downtime 
and  time  up. 

2.  Terminal  monitors  at  each  cluster  will  maintain  terminal  logs  (see.  attached 
Monitors  will  log  the  time  up  for  all  terminals,  time  down,  reason  for 

and  time  back  in  service. 

’When  a  terminal  failure  occurs,  the  monitor  for  that  terminal  will: 

(a)  record  the  fact  on  the  log 

(b)  record  any  symptoms  and/or  the  operation  taking  place  at  the  time 

(c)  communicate  with  central  system  to  try  to  identify  cause  of 
problem 

(d)  record  the  time  the  terminal  is  back  un 


(e)  record  reason  for  downtime. 


PHASE:  III  CTS  FINAL  ACCEPTANCE  TEST  PLAN 
TEST  MONITOR  PROCEDURES  AND  LOG 

General  Instructions  to  Test  Monitors 


The  system  monitor  will  be  located  in  the  computer  room  at  Moran  Hall. 

lie  xifill  observe  and  record  test  activities  talcing  place  there,  as  indicated 

in  te3t  monitor  procedures. 

The.  terminal  monitors  will  be  located  at  Moran,  Greeley  and  Brant  Halls. 
Each  monitor  will  be  assigned  specific  terminals. 

Instructions  to  monitors  are  provided  in  the  Test  Monitor  Procedures  and 
Tog  for  each  test  level.  (Procedures  for  Level  I  are  in  a  separate  volume. 
The  test  controller  will  inform  the  monitors  when  a  test  level  is  beginning 
or  ending. 

Monitors  should  obtain  extra  copies  of  necessary  log  sheets  prior  to 
beginning  a  test  level. 

Monitors  will  submit  their  logs  to  the  test  controller  at  the  end  of  each 
test  day. 


(2)  confinin'.'  to  the  end:  1  hen  rescheduled  to  a  mutual  ly  agreed  up 
'■i<  •JoK-.  The  runnier  ti"-o  i.e-.t;  'pi  a !  0. 

o.  AM  contractor  requirements  for  support  of  the  tesl  will  !c 
coordinated  by  the  COIR. 

PARTICIPANT  ROLES 
PHASE  !  !  I  ACCEPTANCE  TEST  Pi.AN 

1 .  The  government  will  ippoint  a  test  controller.  His  duties  wiM  ir- 
o  I  tide : 

a.  Conducting  the  rest  In  accordance  with  the  test  p'rr. 

I>.  Determining  when  an  element  being  "tested  meets  or  fails  1c  meet 
spec i f ! cat i ons . 

c.  Controlling  the  monitors  at  the  terminal  sites. 

d.  Assuring  that  records  end  logs  are  properly  maintained. 

e.  Conferring  and  consulting  with  contractor  represen : at i vos . 
Bringing  any  unresolved  problems  to  the  attention  of  the  COTR  for  rc'-o- 
I ution . 

2.  The  contractor,  CTE-Sy I  van !a ,  will,  at  their  op+ion,  provide  a  repre¬ 
sentative  who  will  work  with  the  government  test  controller-  In  conducting 
the  tests. 

y,  HumRRO,  the  government's  lest  monitor,  w:ll  monitor  the  test  and  pro¬ 
vide  reports  to  the  Project  ABACUS  C01K  ip  accordance  with  the  condition 
I'M  I'Leir  contract. 

tISAMGS  is  invited  to  provide  observers  tor-  the  ATP  III.  The  observe 
•  iy  p'ovide  comments,  advice,  arid  assistance  to  the  COTR. 

:>.  Dut  ii-s  of  the  terminal  monitors  provided  by  USAS  I GS  and  Pro j or  I  APACU 

in-  !  i :  tii_» : 

a.  -.riiing  of  the  appropriate  data  for  each  level  of  test  for 

tei.-ir  terminals. 

h.  Sun  ‘"vision  of  oer :  onnel  at  te  i  r  ate  I  onod  termi  no  1  :•  to  ensure 
performer  "c  as  required  by  the  test  plan. 

c.  Reporting  and  logging  of  o' I  harm I na I  malfunctions. 

d.  Supervision  and  control  of  terminals  on  site. 
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Gu! nr l infs  rop  phaff  in  acpph ancf  tfst 


'  .  I  ost  Adm  i  n  i  ?lr . :t  ioi; . 

n.  One  person  from  +he  government  will  direct  and  supervise  the  test . 

h.  All  part  ie  I  pants  in  t  ho  tost  v/i  I  I  bo  so  designated  and  their  roles 
•p  f  i  nod  . 

All  personnel,  l.e.,  programmers ,  maintenance  personnel,  instroc- 
*  r  n ,  si  a tents ,  etc.,  not  designated  ns  members  of  the  test  team,  or  speei 
'•  illy  .<«- signed,  will  not  use  the  system  in  any  manner  while  the  test  is 

n  t >rw  ly  . 

*.  tost  plan  should  be  followed,  with  changes  ns  needed. 

All  parsons  performing  test  activities  should  keep  detailed  Ion  of 
•ivlfios.  Log  should  include  time,  response  time,  ATP  paragraph  number; 
if  problem  encountered,  n I  so  conditions,  input,  results  and  retest  rc- 
i  i : cements . 

f.  When  a  problem  or  exception  is  identified.  It  should  be  noted  and 
•i>.-  test  proceed  beyond  that  point.  It  Is  not  feasible  to  diagnose  nil 
'  ins  I n  de+a 1  I  . 

o.  HumRRO  will  supply  a  more  detailed  procedure  +o  follow  for  simulta 
n.'.ais  users  ter, I  so  that  test  activities  can  be  more  carefully  controlled, 
fit'.l  he  approved  by  the  At  my  and  Sylvania) 

h.  If  system  crashes,  entire  ATP  section  major  paragraph  to  he  re- 
+es I ed . 

7.  Accrn+ancn  Test. 

a.  The  Phase  II  Acceptance  Test  Plan  will  be  reviewed  in  total. 

b.  Those  elements  that  were  shortcomi ngs  in  the  Phase  II  test  will  be 
especially  monitored. 

r.  Hie  ATP  w|ll  be  soguenced  by  test  level  as  described  In  the  test 

t't'i.fcedn r<v< . 

d.  All  system  down  lime-  will  be  logged.  If  1  he  logged  down  time 
exceed:'  the  it  in  ?  weeks,  the  test  may  be  at  the  COTR's  option: 

(1)  aborted  and  rescheduled  at  a  date  mutually  agreed  upon,  with  the 
running  time  clock  set  at  0. 


1  or  mi  rt'i  I s  w  i  I  I  be  signed  on  by  120  "si  actants"  and  8  "instructors" 
b-  i»s son  no  Her  in!  provided  by  DSD,  USASIG5. 

Student  response  latency  samples  will  bo  taken  as  in  Level  4  ♦‘•■'d 

Data  from  the  students*  responses  will  be  collected  on  tape  usi_,i 
AV'SR  command. 

(Fs  will  be  signed  on  and  procedures  5  and  6,  Level  4,  re-ex  rut- 
his  will  fake  place  when  it  has  been  determined  -that  the  system  wi*i 
orform  satisfactorily  with  128  "live**  users, 


?.  Fight  "student?"  end  four  "instructors"  divided  between  +ho  four  DCs 
will  sign  on  and  execute  the  program  as  directed. 

5.  An  IP  will  sign  on  and  run  a  compile  of  the  debugged  program. 

4.  An  IP  will  sign  on  and  run  a  compile  of  the  "hugqed"  program. 

5.  Procedures  2,  3,  and  4  will  he  executed  simultaneously. 

LEVEL  4  4  hours  Moran  Hall,  Greoly  Hall,  and  Brant  Hall 

Objecti ves . 

Ascertain  the  system  will  meet  the  response  time  criteria  as  specified  In 
the  design  plan. 

r.-jcedure. 

1.  Assign  sixteen  "students"  and  four  "Instructors"  to  the  system  evenly 
divided  between  the  controller. 

2.  Make  certain  the  simulator  Is  signed  on  for  all  other  terminals. 


3.  On  the  signal  given  by  the  lest  controller,  terminal  monitors  will 
take  random  response  time  samples. 

4.  The  test  controller  will  stop  the  recording  of  samples  and  sign  on  B 
"students"  using  the  USASIGS  prepared  material.  Random  time  sampling  will 
be  restarted  as  in  para  3  above. 

!3 .  The  test  control  ler  will  stop  recording  and  sign  on  4  entry  specialist? 
and  restart  random  sampling  as  in  procedure  3. 

6 .  The  tesl  rordinator  will  stop  recording  and  sign  on  4  IPs  and  restart 
recording  as  In  procedure  3  above.  The  IPs  will  perform  all  normal  function 
except  STIJRUN  and  ASSEMBLE.  At  a  signal  from  the  test  control ler,  the  IPs 
will  execute  STURIJN  and  ASSEMBLE  commands.  Latency  recording  will  restart. 

7.  At  the  conclusion  of  this  test  level,  there  will  be  the  36  "live"  users 
and  92  simulated  users. 


LEVEL  5 


Oh i ect i ves . 


4  hours 


Moran  Hall,  C-rcely  Hall,  and  Brant  Hall 


Ascertain  the  system  will  perform  in  a  satisfactory  manner  with  120  students 
and  8  Instructors  executing  course  material. 


3 


* i”) .  ()  jtn  col  I  f  rcvn  Lo\/r  I  ^  r>n  wM  f  hr'  prorr-^r^vi  thr  .1. <ih  fct 

;ort-Merge  program  tc  lost  the  Sort-Morqe  and  . ■xoi'c:  -o  ! he  tape  ‘r  ,  , 

i.i.’/EI.  2  80  hours  Moran  Ha !  I  ,  Gmely  Half  and  Brant  H->ll 

Tbj  ect  i  vos . 

1.  Ascertain  t  ho  system  will  meet  the  rel  labl  lily  requirement',  c.  ?  * 

*  ine:  for  ten  ft- hour  aays  of  tests  using  the  running  time  merer . 

Ascertain  that,  or,  the  average,  &f  f  terminals  will  operate  70  ei  ‘ : 
to^ai  80  hours. 

f  '-oct'.'hjre. 

!.  The  reliability  test  will  commence  upon  successful  conclusion  r: 

•  vc I  1  test  and  be  run  concurrent i y  with  tost  levels  3,  4 ,  anti  5. 

2.  The  load  simu'ation  program  will  be  loaded  and  all  terminals  ov-.  er> 
those  required  for  Levels  3,  4,  and  5  will  be  activated  under  .lie  tome! 

3.  Mon i tors  at  each  cluster  of  terminals  will  record  any  short rrmi ny;  • 
'ci'uros'.  If  a  failure  occurs,  the  monitor  will  immediately  commu1  ic-'-*, 
.v i  I  h  the  central  system  to  ascertain  if  the  error  is  system  r-  t or.  • 

!  f,d  . 

4.  In  order  that  the  minimum  demands  are  made  or,  manpower  and  time,  if 
the  reliability  test  tlevel  2)  has  not  been  completed  and  all  other  leva 
ero  completed  satisfactorily,  fh^n  Level  2  will  continue  with  only  f.tcr*. 
r-e-sons  present  required  to  insure  the  successful  conclusion  of  the  to 

j  procedure  will  allow  personnel  from  HumRRO,  and  GTt-Sy  I  vania  4c-  >  :■ 
turn  to  their  home  stations  without  wailino  for  conclusion  of  |r>vr<  2 

\  -,;rL  3  ‘1  hours  Moran  Hall 


'  llU  cc.t  i  vos  . 

Ascertain  that  all  CLASS  !  rrvnmr-nds  can  bp  compiled  and  executor. 

Arr-or  fa  i  n  1.,ht  the  CLASS  1  'mailer  will  reconn i?e  and  flag  err'"' 
Mr  i  on  romp  i  I  a  I  icn  . 

■  ,  r.  |  ir  r  . 

1  .  hrr'o  >  or  f>  Sorts  f  fre  \  os  t  pioor-'m  w:ll  bn  i  n  the  svstem.  One  w  i  1  I 
‘  i  he  Da  fa  Guv-  Co  vf  n'f  l-u  ( rrf.  I  wife  "student  s”  rent  stored  ;  another  w  , 
e  ’ll  I  I  y  (lefcuqqed  sen  rod'  d  t  ■<r  I  h  i  rd  will  he  source  r  >dc  w  i  Is 
i  ■  “rrors  i  ne.T'f  •  ■n  . 


TEST  PROCEDURE 

PHASE  1 1 1  ACCEPTANCE  TEST  PLAN  (ATP) 


LEVEL  I  4  hours  Moran  Hall 

Object  I vos. 

1.  Ascertain  tho  full  128  terminal  system  can  be  made  ready  for  operation. 

2.  Ascertain  that  each  type  user  can  log  on  and  perform  those  operations 
allowed.  This  will  be  tested  in  a  single  user  mode  and  simultaneous  user 
mode. 

Procedure. 

1.  System  operator  will  execute  the  system  readiness  test,  ATP  II,  para  3.1. 

?.  System  programmer  will  execute  system  programmer  functions,  ATP  II, 

^ara  3.2. 

3.  An  administrator  will  log  on  and  execute  a  request  (reference  element 
3.4.5  of  the  ATP  ||  not  tested  on  the  first  administration  of  ATP  II). 

4.  Four  IPs  (one  per  DC)  will  log  on  and  execute  ATP  II,  para  3,4. 

5.  Four  "students"  and  four  "Instructors"  (one  each  per  DC)  will  log  on 
and  execute  ATP  II,  para  3.5.1  and  3.5.2. 

6.  Successful  completion  to  this  point  will  verify  the  functions  of  each 
user  can  be  properly  executed  by  the  system  In  a  single  user  mode. 

7.  Simultaneous  user  test  will  be  conducted  In  accordance  with  para  3.6 
of  the  Phase  II  ATP  using  the  following  mix  of  users: 

a.  Four  students  per  DC. 

b.  Two  Instructors  per  DC,  assigned  to  2  students  each. 

c.  One  IP  per  DC. 

d.  One  administrative  user  per  system. 

o.  One  system  programmer  per  system. 

'O.  Successful  completion  of  para  7  will  terminate  the  first  portion  of 
Level  1  test. 

9.  At  i  he  successful  completion  of  Levels  3,  4,  and  5,  the  system  will  be 
set  up  to  process  those  elements  not  tested  In  the  original  Phase  1 1  ATP. 


The  contractor  should  be  required  to  test  thoroughly  any  patches  before  the-' 
are  implemented  on  the  operational  copy  of  the  software.  The  changes  should 
also  be  made  to  the  relevant  software  documentation.  When  the  debugging  and 
patching  phase  is  completed,  the  contractor  should  be  required  to  provlt.a  th 
Army  with  a  complete  reassembled  copy  of  the  software  without  patches. 

h .  Roc  ommend at  io n . 

We  recommend  that  the  system  not  be  accepted  in  its  present  status, 
"allowing  debug  of  the  DC  software  problems  and  conversion  to  the  6B  version 
of  R3X-1J  by  the  contractor  and  CTS  Field  Office,  the  system  should  be  re¬ 
tested  for  overall  performance. 

To  facilitate  testing,  a  new  simulator  should  be  developed  by  the  Army 
•which  more  accurately  represents  operational  conditions. 


halting  (this  is  not  a  non-recoverabl .n  halt).  Software  Systems  Element:  pc 
Executive  or  SC  Software.  Status:  GTE-Sylvania  is  investigating  problen. 

5.1.15  DC  Executive  executes  wrong  TIB:  Under  heavy  traffic  loads, 
the  DC  executive  appears  to  have  a  timing  problem  where  it  attempted  to  ercute 
a  new  student's  TIB  before  the  data  was  read  into  the  TIB  from  disk.  This  re¬ 
sulted  in  the  processor  halting  (non-rec.overable) .  Software  Systems  Element: 

DC  Executive.  Status:  (?IE-Sylvania  is  investigating  problem. 

5.2.0  The  System  Controller  and  Display  Controller  failures  under  heavy 
traffic  conditions  appear  to  be  a  function  of  both  contractor  software  bugs 
in  the  DC's  and  inadequacy  of  the  DEC  RSX-11  operating  system.  The  CTS  Field 
Office  with  the  assistance  of  the  contractor  are  presently  converting  from  the 
present  version  of  RSX-11  (4A)  to  the  more  recent  version  (6B)  which  has  improved 
file  handling  capability.  For  a  discussion  of  this  action,  see  memorandum  dated 
6  August  1976  from  Mr.  Allyn  Evans,  ATTNG-TC-AT-SE,  "Report  of  Trip  to  Proper t 
ABACUS,  Fort  Gordon,  25-30  July." 

5.3.0  System  response  time  in  Instructional  Programmer  mode  was  not 
specified  in  the  contract.  However,  the  delays  experienced  by  IP's  on  the 
current  system  seriously  degrade  the.  productivity  of  the  IP's.  The  ne*7  ver¬ 
sion  of  RSX-11  with  its  improved  file  handling  techniques  will  likely  result 
in  faster  IP  response  time.  If  response  time  is  not  greatly  improved  thereby, 
the  IP  function  should  be  redesigned  such  that  more  of  the  editing  workload 
be  distributed  to  the  DC's,  thus  reducing  the  heavy  traffic  load  on  the  SC. 

5.4.0  Complaints  have  been  voiced  by  the  CTS  Field  Office  systems  personnel 
regarding  the  nature  of  the  program  patches  being  provided  by  the  contractor. 


•  •  •  •  • 
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PHASE  III  CIS  FINAL  ACCEPTANCE  TEST  PLAN 


TEST  MONITOR  PROCEDURES  AND  LOR 

Level  IV:  Response  Time  4  hours  Moran  Hall,  Greeley  Hall,  Brant  Hall 
OBJECTIVES  OF  TEST 

Ascertain  the  system  will  meet  the  response  time  criteria  as  specified  in  the 
design  plan. 


TEST  PROCEDURES 

1  ,  The  test  controller  will  assign  sixteen  "students"  and  four  "instructors" 

to  the  system  evenly  divided  between  the  four  controllers.  The  test  control* e 
will  inform  students  and  instructors  as  to  the  ID  numbers  and  courses  to  use 
for  the  test. 

2..  The  monitors  will  make  certain  that  all  terminals  are  in  operation,  cither 
in  simulation  mode  or  by  live  operators. 

3..  On  the  signal  given  by  the  test  controller,  terminal  monitors  will  take 
random  response  time  samples  in  the  following  manner: 

Beginning  at  a  convenient  place  Jn  the  cluster,  observe  the 
interaction  of  one  "student"  and  his  terminal.  Start  your  stop¬ 
watch  when  the  st.udpnt.  presses  the  NETT  key.  Stop  the  watch 
when  the  system  response  begins  to  appear  on  the  screen. 

Enter  on  your  response  time  log  (sample  attached)  the  time  of  day, 
the  type  of  user  (student,  instructor,  IP,  administrator),  what 
type  of  activity  (scoring  a  multiple-choice  answer,  compiling  n 
program,  etc.),  and  the.  response  time  measured  on  the  stopwatch. 

The  time  of  day  entry  is  important  because  :lt  will,  enable  the  test 
controller  to  determine  the  test  conditions  under  which  that 
response  time  was  obtained. 

1  Move  on  to  the  next  terminal  in  use  by  a  live  student  or  instructor, 

and  repeat  the  above  process.  Continue  to  take  response  time 
measurements  and  make  log  entries  until  test  controller  gives  the 
next  instructions. 

4.  The  test  controller  will  stop  the  recotding  of  samples  and  sign  on  8  "students 
using  the  USASIGS  prepared  material.  Monitors  w:H  make  random  time  sampling 
as  in  paragraph  3  above , 


5.  The  test  controller  will  stop  recording  and  sign  on  4  entry  specialists. 
Monitors  will  restart  random  sampling  as  in  procedure  3. 


Level  IV:  Response  Time  (cont’d) 


6.  The  test  controller  will  stop  recording  and  sign  on  4  IPs  and  restart 
recording  as  In  procedure  3  above.  The  TPs  will  perform  all  normal  functions 
except  STURUN  and  ASSEMBLE.  At  a  signal  from  the  test  controller,  the  TPs 
will  execute  STURUN  and  AS5I2MBLE  commands.  Monitors  will  continue  time 
sampling. 

7.  At  the  conclusion  of  this  test  level,  there  will  be  36  "live"  users  and  92 
simulated  users. 
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PHASE  III  CTS  FINAL  ACCEPTANCE  TEST  PLAN 
TEST  MONITOR  PROCEDURES  AND  LOR 


Level  V  4  hours  Moran  Hall,  Greeley  Hall,  Brant  Hall 

OBJECTIVES  OF  TEST 


Ascertain  the  system  will  perform  in  a  satisfactory  manner  with  3.20  students 

and  8  instructors  executing  course  material. 

TEST  PROCEDURES 

1.  Terminals  will  be  signed  on  by  120  ''students"  and  8  "instruefore,"  using 
the  lesson  material  provided  hy  DSR,  USASIGS. 

2.  Terminal  monitors  will  take  student  response  latency  samples  as  in 
Level  4  test,  using  the  response  time  log  sheets. 

3.  Data  from  the  students'  responses  will  be  collected  on  tape  using  the 
SAVSR  command . 

4.  IPs  will  be  signed  on  and  procedures  5  and  6,  Level  4,  re-executed.  Thi 
will  take  place  when  it  has  been  determined  that  the  system  ri.3  3  perform 
satisfactorily  with  128  "live"  users. 

5.  Terminal  monitors  will  note  any  problems  encountered  by  users  on  the 
terminal  logs. 


